DOCQHBMX BESqHI 



BD 12,^ 888 

AOTHOB 
IITIE 
fob' DATB 
HOTE 



EDBS FBICE 
"DESCBIPTOBS 



iDE^ITIFIEBS ^s!" 



5 IB 003 802 

SBith, Stephen H» . 

Assessing External EvaXuative Infor«ation Needs. 
Jun 76 ' . 1 ' • 

lOp.; Paper presented at the Hational Conference ot ' 
■Open learning and Nontraditional Stqdy (3rd, Lincoln, 
Nebraska, June 15-17, 1976) 

BF-ko. 83 HC-$1. 67 Plus Postage. ' ^ 
♦Edbeational Innovations Evaluation; Eyalu«ion . . 
Crit^ia; *Evaljiatio» Methods; Evaluation Needs; 
Inf orl^tion Needs; Open Education; Post Secondary , 
Educatids^; *Ptogram Evaluation; Spee'cies ° 
External Evaluation; .National Institute of o 
Education - 



xnv< 



ABSTEiJ 

, , A National Institute of Educa-t-ion'-spo&sorrlng project 

stigated how evaluations of nontraditional programs could be 
perforBed tc laxiaize the utility of evaluation findings. This JLed to 
a Conceptual aodel of the etaluation process which included the 
fallowing elements: (1) identification of inforBation needs; (2) 
design, collection, and analysis of inforaation; and (3) 
disseiination of inf oriation. To. deteraine the infoxBation needs for 
the study of -nontraditional programs, 70. policyiakers in 
postaiecondary education were interviewed. The five aajor needs 
identified addressed two issues: (1) whether the needs of th^ learner 
were satisfied; and (2) whether resources were effectively used., , 
(EBBt) • • 
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_A talk'presented at the Third National Gonfere^e on^d|j«^i^ ' ^ 
CO . Learning and Nontraditional' Study, June '14-17, 19^, Lincoln, NE. 

^, Assessing External Evaluative Information- Nee ds \ aSiiftS! 

Stephen M. Smith, Senior Research Associate - |52 illsisi 

Applied Urbane tics, Inc.^ .Washi;ngton\ D.C. sS?g 'Si*-<'S 

. / , ■ ■ . ,p.5<g||Hi 

We are concerned in this session with two broad concepts: jsg" f^flSiS 
nontraditional education and program valuation. More Sg^ |5S^8i| 

specifically, we are concerned with trying to understand 5 ' ^S^iSoj 
. their overrap. I haVe been asked to approach external igjISiS 
program evAluation by relating to you my experience with 
an NIE-spoiWr<d study on exfernal evaluation o£ nontradi- 
tional postsecondary educa^dion programs. 

o 

As you know,'. Congress gave the National Institute of Ed- 
ucation^ (NIE) tijie mission to improve education througJi 
.reseafeh and development. As part of ^.this mission, NIE 
has been concerned- with bo;(;h the evaluation of innovations, 
and .the study of open- learning programs. The former con- 
cern is manifested through their Assessment of Innovative 
Developments program^ and the latter -through their funding ^ 
of the University of Mid-America. ' . 

, Fbr more than a yea^ now, NIE has been developing a program ^k^^. 
v^ich is the offspring pf these two concerns. The focus 
^ the program is the evaluation of open- learning and non- 
traditional programs. The effort is projected to consist ' . 
'•of three parts: (1) pre-design activities; (2) design 
activities; and C3) the . actual evaluation research. 

a ' ■ 

^ NIE selected TEAM Associates, Inc., to perform the first 
• part: the pre-design activities* The activities weref 
based on NlE^s view th^t nontraditional, innovative pro- 
grams require 'nontraditional, innovative evaluations, 
Whetiier this is iii fact true remains -to be seen* Never- 
theless, they felt that innovative education programs are 

d often so 4ifferent in operation and. outcome from traditional 
programs, that evaluatioRs perforjned along-traditional 
^ lines are Irf^^Uvant to the needs and interests of either; 

\j learnefs, -decisiQn-makers or policy makers. This perceived 

irrelevancy in the past has not, however, decreased the 
VM • ^. demand for evaluation of innovations . In fact, innovations H 
are- increasingly subject to questions founded on that \ 
Missourian attitude: "show me 2" 

Q . Our project was concerned, therefore ,'*^^^ith how evaluations 
* of nontraditiohal programs could be performed so as jto 
maximize the utility of the evaluation findings to different 



audiences - audiences external to the program, audiences 
in a ]^sition to influence postsecondary education policy. 
Th^ prHect consisted of two parts. In the first part, 
we reviewesd and aiualyz-ed the evaluation literature and 
interviewed a series of administrators and evaluators. 
We found from this refie.w that evaluative information was 
perceived to lack utility when one or more of the following 
»A>ras prfesent : , 

• "--^ • ' ^ ' 

1 ♦x The information was not presented to the audience, 
iV-e^,,^ the dec isi^jn- maker s , .in a majiner to which 
they. ^re responsive. As one evaluator put it: 
"The pedpje we're interested in influencing, for 
the most part, don't know a chi square from a 
load of coal. If you give them a 40,0-page report, 
it is just not going to g^t read, and if they 
do try and read it, they're going to be turned 
off by .the technical jargon,.". 

2. The information was perceived by the audience to 
lack relevance, timeliness, or credibility, .In 
a review of Federal evaluations, Joseph Whbley 
and/ associates (1970) found that "too many studies 
begir little or no relationship to the critical 
prcigram and policy issues." Commenting on timliness 
Stufflebeam (1971) painted out that for decision- 
making "the best information' is useless if it 
comes too late." The dilemma that plagues eval- 
uations is the competing requirements between 
'timeliness and credibility. John Mann (1969) 
porinted out: "the better the study, the longer 

it takes 'and consequently the less usefulness 
it may have. Conversely, the 'sloppier the pro- 
cedure, the more likely it is to provide infor- 
^ mation on fjuestions- oif interest, even though 
, this datt will be of doubtful validity." 

3. There is a lack of an identifiable, intetested 
«audience . At times, policy issues and decision- 
makers change so frequently that there iS\often 
no interested audience around when a long 'term 
evaluation is completed. At other times, there 
was none to begin. with. 

In order to relate these features hindering utility to 
specific "inadequacies in parts of the evaluation process, 
we developed a conceptual model of this process. This i$ 
shown in Figure 1 in "a much simplified form. l. : 

In the first subprocess, as well. as in the last subprocess. 
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Figure 1 - Evaluat^ion Process / 
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the evaluator acts more as an ad^ 
thaTit^ a researcher. The process 



^or or consultant -rather 
viewed .as an initial 



needs .for information 



series of more and more well defiket 

and data, and a. subsequent series \o£ . more and more analyzed 
results. If the initial need for WWluative information 
by the decision-maker is fulfilled \bjr the communicated 
information at the end of the ijroce^si, then the information 
IS defined as having utility. That\is, there is congruency 
between results and needs "(Smithy 19JS). 

The subprocess which (most leads to the features hindexing 
utility, -that is, ladk of relevance , Itimeliness , etc., is 
the first one, the deMneatioh of the! information needs 
and the communication requirements of the- decision-maker, 

In this subprocess the evaluator is, in effect, performing 
a needs assessment. He is trying to determine the following: 

• \^^rho are the individuals interested in. the evaluati^^? 

• what is the content of their ejalutive information 
needs? "■ 

i . ' 1 ■ 

• for what type of decision is tl^e information needed 
(what resource allocations will be affected?) 

• wh^n is the information needed? , 

9 What level of reliability and validity are neede^ ' 



o what standards of comparison are "requii&d^ 



\ 



o what methods and format arei preferred by the aud- 
,^ ience for presentation of tlhe findings? 

■ i 

As you can see, i^ a set ^xif evaluatlion requirements are 
developed by the decision-maker an^ the evaluator together 
on the ba§is of the aA^wers to thes^ questions, there is 
a high probability tha\ the evaluative information will 
not lack relevance, not lack timeliiless and net lack cred- 
ibility, and it will be presented/ to an interested audience 
in a format that is preferred. | 

In the second part of our project, '\e 
preliminary attempt at performing th!i 
for the projected NIE evaluation of ^ 
Interviews were coridu^cted with appri 
i.e. , individuals who are, formally 
position to :^iifluence postsecondary 
interviewees/were affiliated with Fe 
State legisl/atures, open-learning in 
foundations; Sta'te postseoondary edu 



, in fact, made a 
s needs a^ssessment 
ontraditional programs, 
imately 7f) policymakers 
r inlforlhally, in a 
ducation policy. The 
eral agencies. Congress, 
titutibjis , private ^ i 
ation* systems , pro^/ 
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ional organizations and research groups ' CS^ith and Smith, 1975a). 



were 
shoul 



Throbgh both an- open- and closed-ended inquirV",; the interviewees 



asked to specify those, quesfions that an evaluation 
Id address in order to provide'^^thjm with the most 
useflul information in their administtative, legislative, 
andjpolicymaking activities. We found that five questions 
stood out as leading to information having "the highest 
levels of utility to the- interviewees. These are shown in 
Pigijire 2. , * 

first three of these qyestions are focused on the^ learner , 
latter two are focused on tjie program operation. Any • 
luation which is to be performed will need to emphasize 
data colloction in th^se areas if it is to be useful. We also 
pos ' ' " 

pac 
ti 

Figure 3. NPfl? of these, however, wif^e rated consistently 
at the highest levelsAof utility by majority of inter- ^ 
viewees. . / ^ ^ % 



The 
the 
eva 



ed several questions to interviewees concerning the im- 
t of nontraditional programs on faculty, other institu- 
ns, and the , broader community. These are shown in 



The interviewees' responses w^re aXgtQ divided into several 
categories on. the basis of their pr-^sumed policy role and 
decision responsibilities . We found only two correlations 
between tWt policy role and their information needs that 
cduld be ci^sidered at all conclusive, given the small 
sq[mple in e,ach category: ; ^ 

. - . ■■ ■ -■;/ ^ - . ■ 

1. the evaluative information iieeds of administrators 
^of open- learning programs' appear to be concentrated 

on learner focused questions; and 

2. interviewees affiliated with State government ap- 
pear to rate evaluative information concerning 
program finances more consistently than information 
in other areas at the. highest levels of utility. 

With respect to the othei*^ needs assessment questions, we 
found the following:, 

o for the wide range, of individuals inter- 
^ iMewed'the timing xrequirements varied from 
",as soon as possible" to "the time it takes 
ftp do a valid study.";' " . 

• 'the levels of val-ddity and reliability 

varied from that associated with experimental 
and quasi-experimental research designs^ to » 
that found in journalistic accounts; and 

- *• 

• preferences fof standards of comparison 
spanned the range of possibilities : that is, 
other nontraditional programs, traditional 
programs, no outside comparisons^ . V 
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Figure 2 
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QUESTIONS LEADING' TO yNFORMATION RATED AT ta 
HIGHEST LEVELS OF UTILITY MORE OFTEN THAN OTHERS 



Learner Focused 



Does' the content o£ the curriculum 
addres^ the needs and Wants o£ the 
learners? ^ 



Does the program satisfy learner 
needs, or personal and&ipf ofessional 
interests? "'■-^^^^-SC--' 



Are the procedures for as^sessment of 
learner cpmpetency and achievement 
valid aAd' credible to other /institutions 
or employers? . _ 



Prograip Focused 



Program Finances 



% , How does the cost -effectiveness of 
the program compare with other 
^ programs? / 



* 



Program Process , ^ 

^ •> V . ^ ' \ 

• Wiiich program component?, facilities, 
^ staff or ^finances are essential to 
achievement of the outcomes? 



^ 7 
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FURTHER IMPACT FOCUSEp QUESTIONS 



To what extent can all or part oJ^Tie 

program be adopted .or adapted elsewhere? 

Are the. circumstances or participants jo£ 
• •> 

the program unique? 



What is the impact of changing facif4ty 
roles? . _ 



To what extent does. the program impact 
.on other postsecondary programs, either 
traditional or nontradi^ional? 




One might conclude from these resul 
is some consensus on the focuses of 
the five questionsi ijientioned earlie 
tunately^n the decisioii contexts, 
.is tha^p. few interviewee^ were able 
decisiops for which they would use 
may be xhat many of the. interviewee 
information for reasons other than 
for example, for advocacy or for pe 
with'but further action. 



ts that while there 

evaluation, i*©. , 
r, there is noae unfor- 

What we found, however, 
to identify specific 
the information/ It 
s sought the evaluative 
decisionmaking, perhaps., 
rsonal opinion formation 



Whatever the reasons, this hinders the design of the NIE 
external evaluation because it cannpt be determined which 
of the preliminary specifications are appropriate; the spec- 
ifications for timing, standards of comparison, validity and 
reliability appear unrelated ''to decision Cfontexts. 

The design will also be hindered because well-defined eval- 
uation criteria and indicatoirs still need to be developed 
from the five broad evaluative questions identified. 
For example, in the question "does the program satisfy 
learner needs?", who defines "needs"? ^he teacher? The 
student? or others? Who are the "learners'V, are they in- 
dividuals enrolled in the program or are tKey potential 
enrollees? What is meamt by "program?" - is" if the instruc- 
tional methods or the instructional, content? tjiese questions 
whicfe affect an eva/luation's relevancy, can only be answered 
by the decision maker. " . . , 

ffius, we conclUjde(i that if NIE seeJ^ to perform' external 
eval-uations, and not research s^diesl of nontraditional 
programs, a specific audience with well-defined decision, 
needs must be identified. Otherwise, ^ the eyalUation| findings 
have a high probability of having little utility, not ; 
unlike the^ evaluations of many other innovative educationafl 
programs. (Smith and-Smith, 1975b) ' " j 

Let m? close by suggesting that evaluation is like the 
science of navigatioji, it is but a means tp an end. The 
question, posed much earlier -in history, therefore is even/ 
more pertinent^ tod4y.v~^ 

"What is the use of the scien(j:e of navigation, if we don't 
know where to go?" 
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